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ABSTRACT ^ 

Interim results of the American Institutes for 
Research study on the impact of the ESEA Title VII Spanish/English 
Bilingual Education Program imply that certain inferences can be made 
about the program, inferences which are un justif iab^le. By providing 
only statistical averages on a national sample, the report does a 
serious disservice by failing to distinguish between the effects of 
good programs and^ weaJc programs, and treats b ilingual education as 
funded undef Title VII of ESEA as an undifferentiated uniform whole. 
The report is subject to criticism on' several grounds: (1) weakness 
of Pre-and post-test design over a five-moi^^A period; (2) 
inappropriate use of gain scores to assess effects of experimental 
treatment;. (3) unreliability of teacher assessment of students" 
language ability; (4) inappropriate use of the California Test of 
Basic Skills to assess English reading ability with limited English 
and mdndlingual Span^ish~-speakers; pQssiblx invalidating the results; 
(5) lack of adequate teacher tT^ uing and curriculum during the first, 
five years of Title. VII fcunding ; ^$] distortion of informatipn> 
'defeating the intended purpose of report, i,e,, to provide 

information for policy makers; ana .7) aggregation of students who 
have re'ceived a variety of educational Treatments funded by Title VII 
and who demonstrate varying language ability, (Author/CLK) 
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The pendip.g release of^^tlie^ American Institutes for Research study 
.*''Evaluat ion of the Inpact of ESEA Title VII Spanish/Engl ish .Bilingual 
Education Program'*' gives the illusion that certain types of inferences 
can be made about the . program,- when , in fact, they are not justified. 
:<Thc report, by providing only s.tatistical averages on a national sample, 
docs- a serious disservice by failing to distinguish butv;een tlve effects 
of good prograrro and weak programs, rrnd treats bilingual education as an 
undiff Great iated uri i form whole , which it is not. ' 
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The report is subject to criticism on several grounds:, 

1. V/eakness of pre- and. pes t - tes t design over a'^five month period; 

2. 'Inappropriate use of gain scores to assess effects of 
e>:perir."ient:al treatments; 

3. ' Unreliability of teacher assessment of students' language 

ability; ^ 

4. Inappropriate use of the CTDS to assess English reading ability 
v;ith limited T.iglish and monolingual Spanish speakers, possibly 
invalidat i[ig the results; 

5. Lack' of adequate teacher training and curriculum ■ during the. 
i-rs t~. five, years of Title VII funding: 

6. Distortion of information which defeats the intended purpose 

of the report, -i.e., to provide\in format ion for, policy makers; - 

7. Aggregation of students who have received a variety of 
educational- treatments funded by'Title VII and whose language^ - 

ability ranges- from monolingual English > bilingual '■ — t 

monolingual Spanish conceals the potential effect of bilingual 
education. ^ ' 

Those concerns vnMl be dealt with in detail belov; 



1. According to several noted .psychome trie ians , the use of a pre-post 
test design is- inadequate to determine the significance of change ■ ■ 

between experimental and control groups. This problem is particularly » • 
^acutc here owir.g to the extremely short five month interval between testing 
periods . The research indicates that numerous d if f icult ics arise in 
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the . formulation of questions in terms of gain scores 
limitations" that are inherent in the data,. 



which tend to. cqnceal 



2. Questions about the ^effects of experimental trealimenLs are frequently 
phrased in terms of gains. For example, does treatment^ A result in 
larger, gains tKan treatment B? Do students in Title i VII bilingual classes 
gain more than students in non- treatment classrooms?' Although these t 
questions seem intuitively reasonable, it dpSs not follo\>7 that the 
best approach to answer . them will involve the use of 'measures of change ^ 
as dependent variables, '^la the AIR s tudy., ' "random assignment was not 
poss ible\ since children ar^ ass igned to bilingual programs because of 
their lii\iited Engliish s|>eaking ability, Wi-thout randjom assignrr.ent it is, 
of course\, possible that differences that.may be obserrved in the pos-t test 
score are\the result of pre-existing group diffprencels rather than 
treatment \ef fee ts . " \ 

3. Thp Tredults of the study are questionable when one considers 
that the in\liv.idual student's language ability was assessed by Ehe 
teache-r. Tltis type of assessment is questionable parjticularly -in 
the case of ^ia monolingual teacher evaluating a bilin^:.ual child, ^ 
Research .indicates that teacher assessment is a notoijiously unreliable 
ineasurje of stitydent language proficiency, . | ' 

■ ' - 

4. The sole i\se of the CTCS to assess English reading ability is 
highly" quest io'l^iable particularly ia the case of the ■bilinsualr-Sj)anish- 
do:?.inant and Spanish-dominant children. Although these evaluated projects 
were in the fou'cth and fifth years of funding, the students in these 
classrooms varied in their length of^project participation. Therefore, 
it i.s probable that th^ reading scores of monolingual Spanish speakers* 
and limited English speakers who were recently placed in the bilingual 
classrooms vjould 'have negatively sinewed the test results. The iTiiportant ' 

.. iss ue^.Herc is the\ s tatistical aggregation of test scores without ^regard 
for in-group variatioiiT, It would be equivalent to giving the monolingual 
English students^. the Spanish Prueba; dc Lecture io assess their reading 
skills in an unknovm language . - * 

5. A? though the projects included in tl|is study were in their fourth 
and fifth yesirs of operation, it was nou until 1974 that federal funds 
were appropriated for teacher training or material development centers, 
AIIl's own survey of . the Title VIP projects indicated that they vary 
greatly with regard to teaclfcr expertise, class-hours devoted to 
language instruction in English and Spanish, curricula, conununity 
:^nvolvement , and student language abilities, To compare Title VTI . * \ 
projects as a unit is Jiighly misleading, ' • 
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6. This is not to say that the data base could not provide valuable 
inf orir^at ion concerning bilingual education. The continuation study 

to be completed by x\IR in the Fall of 1977 has the potential to examine 
the important variables which can effect the impact of bilingual- 
education and produce good programs : (1) teacher training and language; 
(2) teacher attitudes; (3) number of years students have been in bilingual 
class.rooms; (4) type of materials used in classron'n,\ etc . It is 
regrettable that the combined e"fforts of the Office c)f Planning, 
Budget and Evaluation (OPBE) and AIR v;ill. not look at ,the important 
issues' of: (1) English .or^l language development; (2) Spanish as a 
seccnd langiiage achieve^.ent : (3) monolingual student gro^*?th; (4) studei^t 
self-concept; (5) attendance and drop-out rates. 

The interim report has analyzed those .results which are of concern to 
OPBE-^not policy makers interested in providing quality education for 
lir.iiLed-English-speaking children. - ^ *' 

7. In general, the type of analyses performed' in the AIR. study 
conci^iais the '"^^ffects of important influences on educational outcomes in 
bilingual education. The fact that this study found little or no 

re lationshit^ bct\;een educational outputs and achievement is highly 
misleading. It. is evident that the** combination of data and statistical 
techniq'^s used are unlil'ely to reveal such relations^iips even wh^en 
they exist;-. The aggregate grouping of s tuden'ts who have received 
a variety of educational treatments v;hich are fupded by Title \^II and 
considered to be bilingual eductxtion and whose langiiage ' ability^^varies 
across the board will •'not provide information' pf value to policy^ makers . 

The ensuing debate over the report in question will do little to 
advance our understanding of how bilingual education af feces children, 
since the Btated conclusions in this interim report; are not warranted 
.by izhe method of analysis used in the report. It will be important to 

v;ait for the completion of •the full study which will provide a more 

detailed analysis of factors 'operative in Id i lingua I programs^. • 
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[For additional, information., contact Dr . Tracy Gray (703) 528-4312] 



